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ABSTRACT 

As part of a new teacher evaluation program initiated 
by the local school board, the Charleston County School District 
(South Carolina) adopted the Assessments of Performance in Teaching 
(apt) as a major evaluation tool to assess the teaching performance 
of annual contract teachers. Since evaluation procedures can 
ultimately lead to teacher dismissal, it was incumbent upon the 
district staff to ensure the appropriateness of the APT and its 
technical quality for a population of teachers wider than those for 
whom the instrument was designed. A study was conducted on 
approximately 250 teachers to examine the inter-observation and 
inter-rater reliability of the APT for various groups of teachers: 
special education teachers, Chapter 1 teachers, elementaf^y, middle 
and high school teachers, black teachers and white teachers. 
Agreement indices were calculated for individual items to identify 
teacher behaviors which reduced reliability and for which observers 
need additional training and practice. Other local concerns addressed 
by the study focused on differences in the ratings of principals 
versus district staff and ratings of observers evaluating teachers 
within their own field of certification versus observers evaluating 
teachers in fields outside their own. (Author) 
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As part of a new teacher evaluation program initiated by the local school 
board, the Charleston County School District adopted the APT as a major eval- 
uation tool to assess the teaching performance of auinual contract teachers. 
Since evaluation procedures can ultimately lead to teacher dismissal/ it was 
incumbent upon the district staii. to ensure the appropriateness of the APT 
and its technical quality for a population of teachers wider than those for 
whom the instrument was designed. 

A study was conducted on approximately 250 teachers to examine the inter- 
observation and inter-rater reliability of the APT for various groups of teach- 
ers: special education teachers. Chapter I teachers, elementary, middle and 
high school teachers, black teachers and white teachers. Agreement indices 
were calculated for individual items to identify teacher behaviors which reduced 
reliability and for which observers need additional treuLning and practice. 
Other local concerns addressed by the study focused on differences in the 
ratings of principals versus district staff and ratings of observers evaluating 
teachers within their own field of certification versus observers evaluating 
teachers in fields outside their own. 
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Responding to the public's outcry for educational accountability, the 
Board of Trustees of the Charleston County School District (CCSD) enacted a new 
policy on teacher evaluation in June of 1982- The intent of this policy was to 
"strengthen" evaluation practices in such a way that incompetent teachers would 
be identified and either remediated, or if remediation failed, dismissed from 
the school system. Board members became attuned to the need for a change in 
current teacher evaluation practices upon heaoring of incidents which caused 
them and the commmunity to question the qxiality of instruction students were re- 
ceiving in the classroom. To assure the community that poor teachers would no 
longer be permitted to continue being employed in the schools, they decided to 
replace the current program with one that could be used to remove teachers who 
lacked basic teaching competencies. This action altered the focus of teacher 
evaluation in Charleston County. Whereas the older program was based upon a 
model of clinical supervision and assumed not only competence but also the need 
for all teachers, regardless of their level of competence, to prepare and imple- 
ment inprovement plans, the new program, upon reqtaest from the School Board, 
was designed to determine whether teachers were competent, i.e., whether they 
possessed skills important for successful performance as a teacher. 

The School Board approached this task logically and recommended to the 

Superintendent that experienced teachers (i.e., those with continuing contracts) 

meet, at minimum, South Carolina's new requirements for beginning teachers. They 

proposed that the instrument developed under Act 187, the "Assessments of 

Performance in Teaching" or APT, be administered to all teachers in Charleston 

County. Administration of the APT became one component of the new teacher 

evaluation program. The APT is described in its manual as follows: 

The APT instrument is divided into five Performance Dimensions. 
Each Performance Dimension is measured through eight to eleven 
statements. The observation statements are dichotomous (yes/no) 
decisions that indicate whether or not a teaching skill or behavior 
was demonstrated during the observation. Specific evidence in the 
form of a statement citing one or more incidents describing the 



demonstration is required. Each Performance Dimension is an 
essential aurea of teaching competence and each must be satis- 
f actor ily demonstrated to successfully complete the APT. The 
assessment is the composite of three observers' ratings (apt 
Manual, page 1) . 

Recent litigation in the area of teacher evaliaation forced district staff 
to consider several issues regarding use of the APT, or any evaluation procedr 
ure, on experienced teachers, under conditions %rfiere the results of that 
procedure could be used to make critical employment decisions. First, the 
coxirts have acknowledged that teachers given tentare or continuing contract 
status are presumsd competent and have legitimate expectations of future 
employment. Consequently, they have protected property interests and can 
insist upon due process procedxires. Evidence for removing experienced teachers 
from. the system must be rather persuasive, and the burden of proof is upon 
school authorities. Second, if more minority teachers are terminated as a 
result of evaluation practices, the school district must demonstrate that 
evaluation criteria are non-discriminatory and related to job perf ormcVice . And, 
finally, it is essential to the validity of the instrument that users demonstrate 
that it yields objective data and can be applied to all teachers in a consistent 
or standardized manner. Failxare to use an evaluation instr\anent for which 
there is sufficient evidence of reliability, as Allen and Jarvis (1983) warn, 
can have significant legal ramifications. 

Since the S.C. State Department of Education developed the APT for 
beginning teachers (i.e., those with provisional contracts)^ it was necessary 
to investigate more thoroughly the extent to which the APT could be applied 
accurately and fairly to experienced teachers by the 130 observers trained to 
^ assist in this evaluation effort. Educators from various segments of the 

district expressed their concerns regarding the reliability of the APT. Trained 
observers expressed concern over training and practice in using the APT instru- 
ment. District staff, principals and assistant principals were trained 
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via videotapes in the administration of the APT and had met the criteria to 
become certified as "endorsed observers/* but they did not have sufficient 
practice, if any, observing teachers in the field. Would the reliability 
demonstrated under controlled "videotaped" conditions be maintained in real-life 
teaching situations? The School Board shared this concern and had questions 
about the reliability of ratings gathered by principals. Board members wanted 
to know whether principals wo\ild tend to be more lenient than district staff 
in their observations of teachers employed in their own school. From a 
similar perspective, district staff were concierned about differences between 
observers rating teachers in their own field of expertise and those rating 
teachers outside their field. Teachers voiced their opinions regaurding the 
observation of teachers in unique types of teaching arrangements, such as 
special education and Chapter I teachers. Could these teachers be evaluated 
objectively and fairly using an instrument which was applied universally to 
all teachers in the district? Finally, there were accusations made regarding 
the discriminatojcy nature of the instrument. Some teachers alleged that 
evaluation would result in the dismissal of proportionally more black teachers 
and erroneously concluded that, if this indeed occurred, the instrument would 
be biased against minority teachers. 

To protect the school district and the rights of teachers, the Evaluation 
and Research (E&R) office recommended that a study be conducted to address 
some of the concerns raised by teachers, administrators and School Board 
menbers. The School Board agreed to the study and to delay until the 
1983-84 school year use of APT data in making decisions regarding the employ- 
ment status of experienced teachers. (This decision coincided with the 
State Department's recommendation to delay the application of the APT to 
decisions regarding the employment of beginning teachers.) Due to the 
limited availability of funds and human resources, the study was designed to 



respond only to reliad^ility issxies^ though at the same time E&R would be able 
to produce for the Office of Teacher Evaluation and Staff Development estimates 
of the percentage of teachers who would fall below the state minimum standard 
and descriptions of teachers* performance on individual competencies and the five 
Performamce Dimensions. Although local educators qiaestioned the extent to 
which some of the 51 ctxapetencies were necessary for successful performance 
as a teacher, validity iasiies were excluded from the study and left to the 
State Department to tackle. 

Questicais addressed by CCSD's Reliability Study were as follows: 

1. What are the score distributions for the APT total score and the five Per- 
formance Dimensions or PDs for teacher subpopulations and racial groups? 
What percentage of teachers fall below the state's minimum standard on the 
total score? 

2. What percentage of teachers demonstrate each of the 51 teaching competencies? 

3. Is tlie APT instrument reliaible for concerned sub-populations of tea^^hers? 
(I.e., do the ratings of two observers evaluating the behavior of a teacher 
at the same time agree? Is the intra-observer variabiliiiy sufficiently small 
to yield reliad^le ratings?) Which competencies appear to contribute to 
measurement error? 

4. Are there differences between the ratings of principals and district staff; 
that is, does one group tend to score teachers lower than the other? 

5. Do district staff observing teachers within their own field of expertise 
rate teachers differently than staff observing teachers outside their field? 
This question was asked only for middle amd high school teachers. 

Evertson and Holley (1981) remind us of the three causes of unreliability: 
(a) unstable phenomena being observed; (b) disagreement between observers about 
what they see occtir; and (c) inconsistency in the way the instrument measures 



teacher behavior. Though we would not be able to distinguish among the causes 
of meastireinent error, if found, we would at least be able to identify where 
problems exist. 
O verview of Study 

A study of the APT was conducted during the 1982-83 school year on samples 
of teachers from various teacher populations - special education, Chapter I and 
"regular" teachers (elementary, middle and high). A group of beginning teachers 
was included for compiurative purposes only. Analyses were performed for these 
groups as well as for racial groups. Different sets of observation patterns or 
"schemes" were used for teacher sanqples. These were counterbalanced and used 
with equal frequency as much as possible. Observers were classified as Princi- 
pals (principals and assistant principals) or District Staff (central staff 
and area superintendents). A third category, "Other," consisting of Principals, 
District Staff and peer teachers who had been endorsed as observers, was 
necessary for some pariis of the study. Observers were then assigned to three 
APT observations which occurred during a two-week interval. 

scheduling and implementation of the study were executed by the Teacher 
Evaluation staff under the guidance of the Evaluation and Research office. 
Observation designs, data entry and data analysis were contracted to Dr. Huynh 
Huynh of the University of South Carolina. 
Study Designs 

Regular teachers. Ninety teachers (54 elementary, 18 middle and 18 high 
school teachers) were selected to participate in the study. A stratified random 
sampling procedure was used to ensure that teachers participating in the study 
were representative of all teachers in Charleston County, with the exception of 
special education and Chapter I teachers. The three types of observers (Princi- 
pals, District Staff and Others) were paired according to the three possible 
combinations, and pairs were assigned to either the first, second or third 
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observation r according to the saaa six observation schemes used in the State 
Department's study. (See Table 1. ) Each observer was assigned to t%»o 
teachers. Fifteen teachers were assigned to each scheme. At the middle and 
high school levels District Staff observers were assigned to one teacher in the 
same field and one teacher in a different field than themselves. 

Special education teachers. Eighteen pairs of Principals and District 
Staff (special education consultants) were assigned to three teachers, yielding 
a total of 54 special education teachers participating in the study. A 
stratified random sampling procedure was used to select teachers representa- 
tive of resource and self-contained placements and the various handicapping 
conditions which reflected the composition of the teaching population. The 
first two observations were conducted individ\iallyr the first by either the 
Principal (scheme 1) or District Staff (scheme 2) , while for third observa- 
tion both observers were present. The first two observations were conducted 
on Tuesday and Thursday of the first week, and the third observation was con- 
ducted on Thursday of the following week. Half the observer-pairs observed 
two teachers according to scheme 1 and one teacher according to scheme 2, 
while the other half observed one teacher according to scheme 1 and two teach- 
ers according to scheme 2. (See Tad^le 2.) 

Chapter I teachers. Twelve language arts and 12 math teachers were 
selected for the study. Pairs of Principal/District Staff observers were 
assigned to two teachers each. Teachers were observed according to the two 
observation schemes used for special education teachers. Each observer-pair 
observed one teacher according to each scheme. (See Table 3.) 

Beginning teachers. Forty eight beginning teachers were selected for 
the study and were observed according to the six observations schemes used 
for regular teachers. Eight teachers were observed according to each scheme. 



ERIC 



6 

9 



Cbstrvtrs v«r« Msigntd to onm t«ach«r only. 
Description of Participating Tuchars 

Although «• anticipated that a total of 216 taachars (118 axperianced and 
48 n«w) would ba participating in all studiaSf tha actual nuai^ar of taachars 
obsarvad was 214. Two taachars (ona ragula^r and ona beginning) had incccDplata 
observation data. The nuaber of teachers froa each group and the nuaber of 
observations are given in Table 4. Table 5 gives a breakdoim of the saaple by 
racef sex, age group and education. 
Presentation of Results 

Since nriaary interest focuses on experienced teachers as a group and on 
differences aMng reg\xlar, special education and Chapter I teachers and between 
black and white teachers, the results of the reliabUity study are presented for 
groups selected f roai the list below according to the qaestions asked in the 
study: 

a. Beginning teachers (47 teachers, 282 observations) 

b. All regular teachers (eleoMntary, niddla, high) (89 teachers, 534 ob* 

servations) 

c. Regular black teachers (36 teachers, 216 observations) 

d. Regular white teachers (51 teachers, 307 observations) 

a. Special education teachers (54 teachers, 216 observations) 
f. Chapter 1 teachers (24 teachers, 96 observations) 

The unit of analysis was observations, rather than teachers. 

Results 

Question Is Score Distributions and Percentages of Teachers Below Standard 

APT scores below standard. Table 6 presents the percentage of total APT 
scores below the state standard of 44 (out of 51 competencies) . These figures 
were used to estiiaate the percentage of teachers who were expected to score less 
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than 44 on th« APT adninist«r«d in 1983-84* Beginning tMch«rs had th« most 
Bcoras telcm standard (26.6%) . A highar parcantaga of black taachars acorad 
balow standard r coa«)arad with whita taachars Spacial aducation and Chaptar I 
taachars obtainad tha highast APT scoras. 

APT scora distributions > Tabla 7 contains tha fraquancy distributions for 
total APT scoras. Cuanilativs parcanugas ara providad for aach tcachar group. 
Vary faw taachars obtainad scoraa of 40 or balow* 

PD scora distributions > 'rha nuaibar of co8«>atancias (and scora range) for 
tha fiva Parformanca DiSMUisions ara: 8 for Planning; 11 for Instruction r 
Kanagwant and Cosvunication; and 10 for Attituda. Cuaulativa parcantagas of 
scoras on tha PDa ara locatad in Tabla 9. Tha sioat noticaabla charactaristic 
of tha fraquancy distributions is tha laOc of variability of tha scoras. For 
axaaipla* only 16.5% of tha scoraa for Nanagasant obtainad by all ragular 
taachars wara 10 or lass of a possibla 11 points, whila 63.5% vara 11. For 
tha othar PDs, about half tha taachars daaonatratad all co«patancias# and tha 
othar half dMonstratad all but ona. Vary fav taachars failad to damonstrata 
two or aoxa ccaqpatancias within a particular PD. This lack of variability was 
not a surprisa, considaring that tha APT assassaa basic taaching co^tancias. 
guastion 2^: Parcantaga of Taachars Who Da»onstratad Each of^ tha 51 Compatancias 

Tha parcantaga of taachars who daacnatratad aach of tha 51 compatancias is 
tha parcantaga of obsarvation shaats on which tha Ltm was codad as danonstratad. 
Tabla 9 prasants this information. Balow is a suaaary of tha compatancias da- 
nonstrated by fawar than 75% of tha taachars (notad by ''X") : 



Coapatancy 



Bag. 



Rag. 
Tot. 



Rag. 
Blk. 



Rag. 

Wht. 



Spac. 
Ed. 



Chaptar 

I 



PD 1: PLANNING 

f * dif farancas planned 

g. objectives assessed 

h. progress provided 



X 
X 
X 



X 



X 



X 



ERIC 



8 



n 



Mq. M9. Urnq. Sp€c. Chapter 
Mq. ^tot, »lfc, Wht, Ed> I 



PP2; HISTWCTIOII 
c. iiMds accQMdat«d 

PO 3; HAKAGPglfT 
(noM) 

PD 4g COHMOMCIiTIOII 

1« written cowunication 



pp St hVmUDK 

d. ImBxninq pcrtonaliMd x X X X 

• • vttlM cownicated X X X X X X 

f • hiaior acknoirlad9«d X X X X X X 



Quattion 2: Haliability of tte APT 

Zndonted mcUom hava baan axtracted varbatia fros a MBorandw 

frott Ruynh (19t3) aiaaarisin? tha raaulte ot tha atudy. 

Tabla 10 raporte tha Inter^^cbaanration and intar-rater raUabilitlaa 
for tha varioua teachar and raoa groupa. For thia tabla, tha indax of 
inter^obaarvaUon raliability vaa takan aa tha conalation batvaan tha 
tuo totel APT acoraa aaaignad on two occaaiona by tha aasa obaanrar. 
For baginning and ngular teachara, thara wara thraa obaarvara (for a 
total of 267 9C0T€ paiT9 and 141 soor^ pain, raapaotix>alyh Aa for 
apacial aducatlon and Chapter Z taachara, thara wara only two obaarvara 
(for a total of 108 aoora pair§ <md 48 aoora pair§, r0$p4otiv0ly} . 

For tha two groupa of baginning and ragular teachara, thara Mra 
12 indicaa of intar-rater raliability. Each i.\vdax waa rapraaantad by 
tha corralatiofia batwaan tha two totel APT tcoraa aaaignad by two 
diffarant obaarvara* Tha ten avaraga inter-rater raliability of Tabla 
10 danotea tha avaraga of thaaa corralation* (Huynh, pagaa 4-5). 

Inter-obaarvation raliability > Intar-obaanration raliability ia an indax 

of tha dagraa to which an obaarvar rataa a teachar consistently fros c!^ obaarvation 

sassion to the naxt» Thasa indicaa ara ascpacted to ba high, though not parfact, 

due to Ainor variations and true inconaistancas in a teacher's behaviors froai 

one day to the next. The indices listed in T^ble 10 are moderately high 

(greater than .60), with a few exceptiona. Highest reliability waa found for 

special education teachers and lowest for Chapter I teachers. Principals 

were most consistent in their ratings of regular teachers and least consistent 

observing beginning teachers. District steff were most consistent in their 
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ratings of special education teachers and least consistent with regard to 
Chapter I teachers. 

Inter- rater reliability. Inter-rater reliability is an index of agreement 
between the total scores assigned by two observers simultaneously rating behav- 
iors of Che same teachers. Indices for beginning and regular teachers were .57, 
while other indices were .69 for special education teachers cmd .30 for Chapter 
I teachers. A difference of .05 was found between indices for black and white 
regular and beginning teachers. Though the difference is minor, the lower 
index for black teachers may be attributed to the significantly fewer black 
teachers observed. 

The Reliability Training Program developed to train and certify APT obser- 
vers sets a minimally acceptable reliability standard of .80. None of the 
reliability estimates obtained in this study reached that figvire. The esti- 
mates for Chapter I teachers are extremely low compared to those found for other 
teacher groups and suggests problems with using the APT for this group of 
teachers without some further investigation. 

The overall reliability of the total APT scores and the associated stan- 
dard error of measurement (SEM) are docxamanted in Table 11. 

In this table, the standard deviation (SD) was obtained by combining 
all total APT scores for each teacher group in one sarole. For each tea- 
cher group, reliability was taken as the average of all the inter-observa- 
tion and inter-rater correlations. (For beginning and regular teachers, 
there are 15 such correlations. As for special education and Chapter I 
teachers/ these correlations number at 6.) The standard error of 
measurement wais computed via the formula 

SEM « SD (1 - reliability) . 

Table U also reports the reliability and standaard error of 
measurement for all teachers. The overall reliability (.589) was 
derived by taking the weighted average of the reliabilities of the 
four teacher groups with each reliability weighted by the number of 
teachers in the group. The overall standaard error of measurement 
(2.19) was computed via the formula listed in the last paragraph 
(Huynh, page 5.) 

Based upon the results, the standaurd error of measurement can be 
estimated at two for the APT. The overall reliabilities again identify 
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potential problems with the use of the APT for Chapter I teachers. 

Item reliabilities > The extent to which observers agreed on their 

ratings of individuals items is documented in Table 12. 

To combine all the data for the pxirpose of examining the 
reliability of each item (skill) , the observations made by the 
category ••Other** were deleted from the two groups of beginning 
and regular teachers. Thus, in the combined data, there were four 
observations made on each item for each teacher. Each observation 
was coded as 0 (no evidence of the skill) or 1 (evidence of the 
skill) . 

For each item, the reliability was taken as the percentage of 
times in which two separate observations made by the category •*Other" 
were both zero or one. Thus the item reliability was taken as the 
raw agreement index taken over the observers and for the group of 
teachers under consideration. 

...In the interpretation of the item reliability, please note 
that its chance level is .50. This level will occur if all observers 
remdomly assigned their scores to the items (Huynh, page 5) . 

In general, raw agreement was lowest for competencies demonstrated by 
fewer than 75% of the teachers. No doubt the greater variability in the 
degree to which teachers demonstrated these behaviors contributed to the lower 
agreement indices. 

Question 4_: Differences Between Principals and District Observers 

Table 13 lists the mean and standard deviation and the percentage of 
cases below 44 for the total APT scores assigned by the principals and district 
observers for each teacher group. 

Overall, the mean difference between the total APT scores 
assigned by the principals and by the district observers was .35 
on the 51-score APT scale. In terms of the percentage of 
observations below the state passing score of 44, the difference 
between the two groups of observers was one percent. Judging 
from both the mean and the percent of cases below 44, the data 
indicated that district observers tended to score lower (be 
"harder") than the principals when all teacher groups were 
combined. This trend, however, was not consistent across the 
four individual teacher groups (Huynh, page 8) . 
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Question 5: Differences Between In-Field and Out-of -Field Observers 

Below are listed the mean, standard deviation and percentage of cases 
below 44 for the total APT scores assigned by the in-field and out-of-field 
district observers. The data were compiled from the group of middle and high 
school teachers. 

Number of Number of Percent 
Observer Teachers Observations Meam S.D. below 44 

In-field 18 36 47.06 3.22 11 

Out-of-Field 18 36 46.36 4.04 22 

The data indicate that district staff observing teachers within their 
own field assigned higher scores (thus failing less teachers) than when ob- 
serving teachers outside their own field. Although the mean scores are 
similar/ out-of-field observers failed am additional four of the 36 observations. 

Conclusions 

The reliaO^ility indices found in this study were much lower than the index 
of .80 used to endorse APT observers. Data comparing raw agreement indices 
with percentages of teachers demonstrating each competency suggest that the 
inter- and intra-rater reliatbility of the APT observations would be much 
lower if there were greater variaQjility in teachers* performance on the APT. 
We can also project that the APT would be more reliable for high-scoring 
teachers and less reliable for low-scoring teachers • This trend has strong 
implicatit ,is for use of the APT in employment decisions , since teachers 
scoring below the passing standatrd will be those considered for dismissal 
from the system. It would be wise, therefore, to exercise caution in using 
APT scores for sijumnative decision-making without either demonstrating 
the reliatbility of APT scores for targeted teachers or accumulating additional 
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evidence of incompetency. 

Data comparing different categories of observers (i.e., Principals vs. 
District Observers; In-Field vs. Out-of-Field Observers) show minor differ- 
ences between ratings. However, the cpiestion of whether or not observers 
would be less reliable vrtien assessing teachers on future occasions or without 
another obsexrver present should not be ignored in this particular situation 
where observers, mostly principals, axe forced into conflicting evaluative and 
supportive roles. One way for principals to reconcile their new evaluative 
function with their well-established and well-accepted supportive function 
is to be more lenient in their ratings, i.e., when in doubt give teachers 
credit for demonstrating a particular competency. In fact, preliminary data 
on CCSD*s 1983-84 teacher evaluation program indicate that the distribution of 
APT scores is much more negatively skewed than last year's. There is also 
evidence that a substauitial minority of observers have '•favorite competencies'* 
and are more likely to deny credit for them to a greater degree than other 
behaviors. 

The new wave of accountability, coupled with the expci^ding literature 
on teacher and school effectiveness, will encourage more and more states and 
school districts to evaluate teacher performance through classrocc; '^>"j>ervational 
techniques. These assessment procedures, though not new to educational re- 
seaxchers, are quite novel to school principals, the principle evalviators of 
teachers. Not only must these individuals deal with role conflicts, but they 
also must leaorn, practice and perfect a new method of teacher evaluation and 

'use observation procedures in a consistent and reliable miuiner. When teacher 

t 

evaluation is based upon a high-inference rating system, such as the APT, which 
requires a greater amount of interpretation compaired with low- inference 
measures, it is critical that users of observation instruments mandate that 
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observers successfully participate in a reliaibility training program. In 
addition, users should also: (a) allow sufficient lead time before imple- 
caenting the evaluation system so that observers can practice their observation 
skills; (b) continue to periodically collect data on rater reliaibility after 
preliminary studies have been completed; and (c) redefine and re-clarify 
descriptions of teacher behaviors and competencies contained on the observa- 
tion instrxHoent to reduce, as much as possible, subjectivity of the instru- 
ment, thereby increasing rater reliaJaility . 
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Table 1 

APT Reliability Observation Scheme for 
Experienced and Beginning Teachers 







Observation 




Scheme 


First 


Second . . ^ 


Tbird 


1 


Principal 


Principal 


District Staff 




Other 


District Staff 


Othar 


2 


Principal 


Principal 


District Staff 




District Staff 


Oth«r . . 


Othar 


3 


Principal 


District Staff 


Principal 




District Staff 


Othar 


Othar 


4 


Principal 


District Staff 


Principal 




Other 


Othar 


District Staff 


5 


District Staff 


Principal 


Principal 




Other 


District Staff 


Othar 


6 


District Staff 


Principal 


Principal 




Other 


Othar 


District Staff 
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Table 2 

APT Reliability Observation Patterns for 
Special Education Teachers 



Observation 

Observation • 



Pattern Teacher Scheat First Second Third 

A 11 Principal District Principal/District 

2 2 District Principal Principal/District 

3 1 Principal District Principal/District 

B 12 Dis^ict Principal Principal/District 

2 1 Principal District Principal/District 

3 2 District Principal Principal/District 
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Table 3 

APT Reliability Observation Schemes for 
Chapter I Teachers 



Observation 

Scheme First Second Third 

1 Principal . District Principal/District 

2 District Principal Principal/District 
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Table 4 
Sample Description 



Number of Number of Observations* Total NuBiber of 

Grot^) Teachers Per' Teacher Observations 

Regular 89 6 534 

Special 54 4 216 
Education 

Chapter I 24 4 96 

Total 167 846 
Experienced 

Beginning 47 6 282 

TOTAL 216 1,128 
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Tabl« 5 



Description of Participating Teachers: Number and Percentage 
(Within Teacher Group) According to Biographical Variables 



















TOTAL 






Biographical 


Regular 


Spec. 


Ed. ' 


Chapt. 


I 




EXPERIENCED 


BEGINNING 


Variable 


# 


% 


# 


% 


# 


% 




* 


% 


# 


% 


Race 
























Black 


36 


41% 


12 


22% 


15 


63% 




63 


38% 


1 


2% 


White 


51 


59% 


40 


74% 


4 


17% 




95 


57% 


43 


91% 


No Data 


2 


2% 


2 


4% 


5 


21% 




o 






6% 


Sex 
























Male 


14 


16% 


7 


13% 


1 . 


4% 




22 


13% 


7 


15% 


Female 


75 


84% 


46 


85% 


22 


92% 




143 


86% 


38 


81% 


No Data 


0 


- 


1 


2% 


1 


4% 




2 


1% 


2 


4% 


Age 
























20-25 


3 


3% 


9 


17% 


1 


4% 




12 


7% 


24 


51% 


26-30 


14 


16% 


16 


30% 


1 


4% 




32 


19% 


10 


21% 


31-40 


38 


43% 


21 


39% 


9 


38% 




68 


41% 


7 


15% 


41-50 


23 


26% 


2 


4% 


1 


4% 




26 


16% 


3 


6% 


51 or more 


11 


12% 


5 


9% 


9 


38% 




25 


15% 


0 




No Data 


0 




1 


2% 


3 


13% 




4 


2% 


3 


6% 


Education 
























Bachelor Degree 


50 


56% 


26 


48% 


14 


58% 




90 


54% 


36 


77% 


Master Degree 


30 


34% 


26 


48% 


6 


25% 




62 


37% 


8 


17% 


Master Deg. 6 30 hrs 


5 


6% 


0 




3 


13% 




8 


5% 


0 




Doctorate 


1 


1% 


2 


4% 


0 






3 


2% 


0 




Bus., Cler., Voca* 


2 


2% 


0 




0 






2 


2% 


1 


2% 


Other 


1 


1% 


0 




0 






1 


.5% 


0 




No Data 


0 




0 




1 


4% 




1 


.5% 


2 


4% 
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T«bl« 6 



P«rcttnt«g« of Tot«l APT Scores Btlow Revised state Standard (44) 



Teach* r Group 


ALL TEACHERS 
No. . NO. 
Tchrs. Obs. 


W 

% 


■LACK TEACEERS 
No. No. 
Tchrs. Obs. % 


WHITE TEACHERS 
No. No. 
Tchrs. Obe. % 


Beginning Teachers 


47 


282 


26.6 


1 


6 


33.3" 


43 


258 


27.5 


Experienced Teachers 




















Regular 


89 


534 


14.2 


36 


216 


21.3 


51 


306 


9.2 


Special Education 


54 


216 


7.9 


12 


48 


10.4 


. 40 


160 


7.5 


Chi^ter I 


24 


96 


5.2 


IS 


60 


6.7 


4 


14 


6.3 


TOTAL 


167 


846 


11.9 


63 


324 


17.0 


95 


480 


8.5 



*Due to missing data on raca, tha nuabar of vhlta taachars and tha nunbar of black 
ttachars do not add xxp to tha total numbar of taachars. 



*^8ad on only taachar (with six APT scoras) . 
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Tabl« 7 

Frequency Distributions for Total APT Sccrss: 
Cumulative P«rc«ntS9€s 





Regular 


Regular 


Mgular 




Chaot. r 


Total Score 


Total 


lUck 


White 


So«C. Ed. 


27 


0.2 


- 


0.3 






2t 


0.2 


- 


' 0.3 


- 




• 29 


0.2 


- 


0.3 


- 


• 


30 


0.4 


- 


0.7 


- 




31 


0.4 


- 


0.7 


O.S 


- 


32 


0.4 




0.7 


0.9 


- 


33 


0.4 


- 


0.7 


0.9 


— 


34 


0.6 


- 


1.0 


0.9 


— 


3S 


0.9 


0.9 


1.0 


0.9 


- 


3(> 


l.S 


2.3 


1.0 


1.4 


- 


37 


1^9 


3.2 


1.0 


2.3 


- 


2% 


2.2 


3.7 


1.3 


2.8 


- 


39 


3.2 


6.0 


1.3 


2.8 


- 


40 


4.1 


6.9 


2.0 


3.3 




41 


6.6 


10.6 


3.6 


3.3 


2.1 


42 


10.7 


16.2 


6.5 


5.1 


4.2 


43 


14.2 


21.3 


9.1 


7»9 


5.2 


44 


19.5 


28.2 


12.7 


11.6 


12.9 


4S 


27.5 


38.0 


19.9 


14.4 


22.9 


46 


37.5 


51.4 


27.8 


27.4 


39.6 


47 


47.2 


63.0 


35.9 


41.4 


63.5 


48 


59.9 


77.8 


47.7 


55.8 


81.3 


49 


74.9 


85.6 


68.0 


75.3 


96.9 


50 


90.8 


95.4 


87.3 


92.1 


99.0 


51 


100.0 


lOO.O 


lOO.O 


100.0 


100.0 
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Ttmqrfimncy Distributions for PD Scorss: 
OwuIstiYs Psrctntagss 



PsrfoxBiincs 


PD 




lt«9ul«r 










Scors 


Tsui 




Whit« 


Spsc. Ed. 


Chspt. Z 


I • FX«nnxn9 


X 








ft c 
0.5 








ft T 


y A 

1. 4 


ft ^ 


ft C 
U. 5 






J 


1»9 


J» 2 


1 ft 

1.0 


2. J 


\ ft 






4. J 




3.3 


£ A 
^.0 


2. 1 




K 


ll.l 


17. o 


7.2 


9. 3 






6 


27.5 


37.5 


19.9 


26.0 


31.3 




7 


51.3 


€4.4 


41.8 


5t.l 


67.7 




m 


t AA ft 


H ftft ft 

100*0 


^ AA A 

100 .»0 


^ Aft A 
100.0 


1 ftft ft 
100. o 


IZ. Inttmctlon 


5 






A *V 

0.7 


A S 

0.5 






h 


0.4 




0.7 


0.9 






1 


1.5 




1.3 


1.4 






c 


4.5 


••3 


2.0 


2.1 








15.7 


25.5 


9.5 


9.3 


7. j 




10 


4A.4 


53.7 


33.7 


37.7 


34.4 




11 




lOO.O 


% A A A 

100.0 


« A A A 

100.0 


1 AA A 

100.0 


III. tunsgssMnt 


I 


0.4 




0.7 








2 


0.4 


- 


0.7 


- 


- 






0.4 




0*7 








4 


ft A 

0*4 




A *• 

0.7 








5 


A A 

0»4 




A *¥ 

0.7 








o 


0.7 




1. 3 








7 


0.9 


A S 

0.5 


1.3 








8 


2.8 


2.3 


2.9 




3.1 




9 


6.1 


6.9 


5.6 


2.3 


3.1 




1 A 

10 


lo.5 


17.1 


16.3 


10.2 


16.7 




11 


« AA A 
100* 0 


% AA A 

100.0 


^ AA A 

100.0 


« aa a 

100.0 


1 A A A 

100.0 


IV. coHBunicAcion 






Ml 




A f 

0.5 






n 

/ 


A ^ 

0.7 


A A 

0.9 


A ^ 

C.7 


2.3 






ft 

o 


2.4 


4.6 


% A 

1.0 


3. 3 








15.5 


21.3 


11.4 


11.6 


1 A ^ 

10.4 




lU 


52*4 


iC A A 

69.9 


^ A A 

39.9 


45.6 


58. 3 




11 


100.0 


100.0 


100.0 


100.0 


100. 0 


V. Attitude 


2 


0.2 


0.5 










3 


0.4 


0.9 










4 


1.1 


2.3 


0.3 




1.0 




S 


3.9 


6.9 


1.6 


3.3 


2.1 




6 


10.1 


14.4 


7.2 


4.2 


6.3 




7 


19.3 


21.8 


17.6 


15.8 


28.1 




8 


40.1 


45.4 


36.3 


31.6 


59.4 




9 


68.5 


75.9 


62.7 


65.1 


88.5 


1 


10 


100.0 


100.0 


100.0 


100.0 


100.0 
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TabU d 

P«rc«ntJi9« o£ <»««rvation Sh««t« on Which T«ach*x.» 
Ocaonrtratcd Each Con^tency 



APT 
Cowpaf ncy 



Bagin. 
Taachars 



Ragular 

Total 



Ragular 
Black 



Raoxilar 
wiiita 



Spec. 
Ed. 



Chapt. 
I 



PD 1: PLUmiMS 



a. 

b. 
c. 
d. 
a. 
£. 

g. 

h. 



OUtCOMS stAtad 
objactivas conpatibla 
procaduzas atatad 
atudants involvad 
aatariala statad 
diffarancaa plannad 
ohjactlvaa aaaaasad 
prograas racordad 



81.2 
97.2 
89.0 
99.6 
89.4 
68.1 
57.8 
61.3 



88.0 
97.8 
93.1 
99.6 
90.1 
72.5 
76.4 
85.0 



86.1 
97.7 
89.8 
99.1 
86.1 
62.5 
74.1 
75.0 



90.5 
98;.0 
95.1 
100.0 
93.1 
79.7 
78.4 
91.5 



88.4 
98.1 
90.3 
98.6 
88.0 
82.4 
70.8 
81.0 



87.5 

97.9 

87.5. 

97.9 

83.3 

84.4 

71.9 

81. i 



PO 2: INSTRUCTION 

a. bagan proaptly 

b. objactivaa addraaaad 

c. naada accoaodatad 

d. Intaraat atiaulatad 
a. approachaa varlad 

f . slxaa variad 

g. activa oppoxtunitiaa 

h. application opport. 

i. infoxnation obtainad 
j. prograaa provldad 

k. phyaical arzangaaant 



98.9 
98.2 
64.5 
77.7 
98.9 
85.1 
98.6 
94.7 
97.2 
98.2 
97.2 



98.9 
98.5 
73.4 
89.0 
100.0 
87.6 
99.4 
97.4 
96.8 
98.3 
96.8 



99.5 
99.1 
60.6 
84.7 
100.0 
82.4 
99.1 
95.8 
96.3 
96.8 
96.3 



98.4 
98.0 
81.4 
91.5 
100.0 
91.5 
99.7 
98.4 
97.1 
99.3 
97.1 



99.1 
99.1 
82.9 
83.8 
93.6 
90.3 

100.0 
98.6 
96.3 

100.0 
99.1 



97.9 

100.0 
90.6 
82.3 

100.0 
93.8 
99.0 

100.0 
97.9 
96.9 

100.0 



PO 3: MANAGEMENT 

a. bahavior aatabliahal 

b. firm anforcamant 

c. procadxiral confidanca 

d. inatruction continued 
a. disruptions addraaaad 

f . codas anforcad 

g. inattentive involved 

h. special assistance 

i. strategies adjusted 
j. patieittf poised 

k. fair, iapartial 



98.6 


98.5 


99.1 


92.9 


98.1 


99.1 


95.7 


99.1 


99.1 


87.9 


95.1 


94.0 


93.3 


96.6 


96.3 


93.3 


96.8 


98.6 


85.1 


96.3 


95.8 


97.2 


96.6 


96.8 


95.0 


97.8 


98.6 


98.9 


98.3 


97.7 


97.5 


97.8 


98.1 



98.0 


99.5 


99.0 


97.7 


100.0 


97.9 


99.0 


99.1 


99.0 


95.8 


97.7 


91.7 


96.7 


99.5 


97.9 


95.4 


97.2 


97.9 


96.4 


98.1 


97.9 


96.4 


99.1 


100.0 


97.1 


98.6 


97.9 


99.0 


99.1 


100.0 


97.7 


99.5 


97.9 



(continued) 
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Percentage of Observation Sheets on Which 
Teachers demonstrated Each Competency 



Table 9 (continued) 



APT' 
Competency 



Begin. 
Teachers 



PD 4: CCMKUNICATION 

a. instructional plan 

b. logicid. sequence 

c. understandable level 

d. explanations restate 

e. illust. demonstrated 
£. knowledgeable auth. 

g. information accurate 

h. legible writing 

i. written coammicat. 
j. oral conmrunication 
k. speech quality 



83.0 
98.2 
98.9 
98.6 
96.4 
98.6 
98.6 
90.4 
48.9 
99.6 
99.3 



Regular 
Total 



90.3 
99.8 
99.4 
97.4 
97.4 
99.1 
96.4 
94.8 
58.4 
97.0 
98.9 



Regular 
Black 



85.6 
'100.0 
100.0 
98.1 
96.8 
98.6 
96.8 
93.5 
40.3 
94.4 
99.1 



Reguleu: 
White 



93.5 
99.7 
99.0 
96.7 
97.7 
99.3 
96.1 
96.1 
71.6 
98.7 
98.7 



Spec. 
Ed. 



Chapt. 
I 



86.1 
100.0 
98.1 
99.5 
94.9 
99.1 
97.2 
96.3 
66.2 
99.1 
99.5 



86.4 
99.0 
99.0 
100.0 
100.0 
100.0 
96.9 
95.8 
54.2 
100.0 
100.0 



PD 5: ATTITUDE 

a. courtesy modeled 

b. positive reinforce. 

c. expression encouraged 

d. learning personalized 

e. supportive correction 

f . reasons given 

g. value conminicated 

h. enthusiasm cooBunic. 

i. op-sn-mindedness 

j. huntor acknowledged 



98.2 
94.3 
83.3 
68.8 
94.7 
81.2 
59.9 
75.2 
99.3 
61.7 



97.9 
94.2 
90.1 
74.7 
95.7 
86.1 
64.8 
83.3 
99.6 
69.9 



99.1 
89.4 
89.8 
69.0 
95.4 
82.9 
57.4 
82.9 
99.5 
66.7 



97.7 
97.4 
89.9 
79.4 
95.8 
88.9 
70.6 
83.3 
99.7 
71. S 



99.1 
99.1 
89.4 
78.7 
99.1 
88.9 
72.7 
88.9 
100.0 
64.8 



100.0 
92.7 
86.5 
66.7 
90.6 
89.6 
70.8 
75.0 

100.0 
42.7 
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Table 10 



Inter-Ob servat Ion and Inter-Rater Reliability 



Teacher 
Group 



Nuaber 
of 

Teachers 



Inter-Observatlon Reliability 



Principals District 



Other 



Average 
Inter-Rater 
Reliability 



Beginning teachers 

Regular teachers 

Special education teachers 

ehapter I teachers 
Beginning & Regular teachers 

Black 
White 

Special & Title 1 teachers 

Black 
White 



47 
89 
5A 
2A 

37 
9A 



27 



.506 
.713 
.689 
.542 

.787 
.614 



.636 
.703 



.664 
.590 
.777 
.389 

.672 
.634 



.714 
.717 



.622 
.684 



.734 
.650 



.574 
.572 
.687 
.298 

.557 
.604 



.541 
.671 



29 



30 
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Table II 



Overall Standard Deviation, Reliability and 
Standaurd Error of Measurement 



Number of 

T^achT Group Teachers SD ReUability SEM 



Beginning teachers 


47 


3.646 


.578 


2. 


37 


Regular teachers 


89 


3.402 


.590 


2. 


18 


Special education teachers 


54 


3.054 


.702 


1. 


67 


Chaqpter I teadiers 


24 


1.949 


.354 


1. 


57 


Black teachers 


64 


3.263 


.589 


2. 


09 


White teachers 


138 


3.511 


.633 


2. 


13 


All teachers 


214 


3.410 


.589 


2 


.19 
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Table 12 



Average Raw Agreement Indices* for 
Each Competency 



APT 
Competency 



Begin. 
Teachers 



Begrilar 
Total 



Pegulau: 
Black 



Regular 
White 



Spec. 
Ed. 



PD 1: PLAHNING 

a. outcomes stated 

b. objectives compatible 

c. procedTires stated 

d. students involved 
materials stated 

f. differences planned 
objectives assessed 
h. progress recorded 



.84 

.96 
.84 
.99 
.33 
.73 
.77 
.72 



.86 
.97 
.90 
.99 
.86 
.80 
.76 
.87 



.82 
.97 
.86 
.99 
.82 
.75 
.72 
.81 



.90 

,98 

.93 

1.00 

.88 

.84 

.80 

.90 



.84 

.96 
.89 
.98 
.83 
.83 
.73 
.81 



PD 2: INSTRUCTION 

a. began proji?>tly 

b. objectives addressed 

c. needs accomodated 

d. interest stimulated 
approaches varied 

f . size varied 

g. . active opport\inities 

h. application opport. 

i. information obtained 
progress provided 

Jc. physicaLL arrzmgement 



.99 
.99 
.72 
.69 
.98 
.79 
.97 
.91 
.95 
.98 
.95 



.98 
.97 
.75 
.88 
1.00 
.84 
.98 
.96 
.95 
.97 
.95 



1.00 
.99 
.70 
,87 

1,00 
.80 
.97 
.94 
.94 
,95 
.94 



PD 3: MANAGEMENT 



a. 
b. 
c. 
d. 
e. 
f- 

g- 
h. 
i. 

j. 
k. 



behavior established 
firm enforcement 
procedural confidence 
instruction continued 
disruptions addressed 
codes enforced 
inattentive involved 
special assistance 
strategies adjusted 
patient, poised 
fair , impeurtiad. 



.97 
.92 
.96 
.88 
.89 
.89 
.85 
.94 
.96 
.98 
.96 



.98 
.99 
.99 
.92 
.96 
.96 
.95 
.95 
.96 
.98 
.96 



.99 
.99 
1.00 
.90 
.96 
.99 
.97 
.94 
.97 
.97 
.97 



.97 
.96 
.78 
.89 
1.00 
.88 
.99 
.97 
.95 
.98 
.96 



.98 
1.00 
,98 
.93 
.96 
.94 
.93 
,96 
,96 
.99 
.96 



.98 
.99 
.84 
.77 
.98 
.87 

1.00 
,57 

.94 
1.00 
.98 



.99 
1.00 
.98 
.96 
.99 
.95 
.97 
.98 
.97 
.98 
.99 



*The raw agreement index is the percentage of observations for which raters agreed on 
their ratings of an individual item. The index can range from 0 to 1.00, with 0 indicating 
no agreement on any observation and 1 indicating agreement on all observations. Since 
chance agreement is .50, the index actually ranging from .50 to 1.00. 
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Average Bmm Agreement Indices 

for Each Con^etency 
Table 12 (continued) 



APT 

Comcetency 


Begin. 
Teachers 


Regular 
Total 


Rtgular 
Black 


Pegular 
White 


Spec. 

Ed. 


Chapt. 
I 


PD 4: COMKUNICATION 

a. instructional plan 

b. logical sequence 

c. understandable lavel 

d. explanations restated 

e. illust. deaonstrated 
£• knowledgeable auth. 

g. information accxirate 

h. legible writing 

i. written cosnunication 
j. oral cosBunication 

k. speech quality 


.81 
.96 
.98 
.97 
.97 
.98 
.98 
.86 
.61 
1.00 
.99 


.36 
1.00 
.99 
.97 
.95 
.98 
.93 
.91 
.66 
.96 
.98 


.81 
1.00 
. 1.00 
,99 
.92 
.97 
.94 
.89 
.67 
.95 
.99 


.89 
1.00 
.98 
.96 
.97 
.99 

.93 
.65 
.97 
.98 


.80 
1.00 
.98 
.99 
.92 
.99 

. 

.94 

.76 
.99 
.99 


.83 
.98 
.98 
1.00 
1.00 
1.00 
.95 
.92 
.58 
1.00 
1.00 


PD 5: AWITUDB 

m. courtesy laodeled 
b« positive reinforce. 

c. expression encouraged 

d. learning personalised 

e. supportive correction 
c. rcuons '91 wn 

9. valu* coonxmicatcd 
h. «nthuaiasa coonunic. 
1 . opcn-mindadn* ss 
j. huBor acknowledged 


.99 
.94 
.76 
.63 
.96 
.73 
.63 
.69 
.99 
.64 


.97 
.92 
.85 
.69 
.93 
.82 
.65 
.80 
.99 
.71 


1.00 
.87 
.88 
.62 
.89 
.79 
.62 
.79 
.99 
.66 


.96 
.96 
.82 
.74 
.95 
.86 
.69 
.81 
.99 
.75 


.98 
.98 
.81 

.75 
.99 
.82 
.68 
.88 
1.00 
.73 

1 


1.00 
.91 
.74 
.60 
.85 
.82 
.67 
.67 

1.00 
.56 
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TabU 13 

Di£f«renc«s in Mean, Standard Deviation 
and Parcant Balow 44 for Principals and District Obsarv^rs 



T««ch«r 


M\anber of 
Observations 


Observer 


Mean 


SD 


Percent 
below 44 


Baginning teachers 


94 


District 


45.14 


3.61 


22.3 




Principal 


45.48 


3.44 


25.3 


Regular teachers 


178 


District 


46.74 


3.29 


15.2 






Principal 


47.23 


3.56 


12.9 


Special education teachers 


108 


District 


47.26 


3.18 


9.3 




Principal 


47.72 


2.91 


7.4 


Chapter I teachers 


48 


District 


46.90 


2.02 


6.3 




Principal 


46.56 


1.88 


4.2 


All teachers 


428 


District 


46.54 


3.30 


14.3 






Principal 


46.89 


3.32 


13.3 
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